[Remote Store] Fix stats reporting for multistream downloads. #10402

Rishikesh1159 · 2023-10-05T13:50:56Z

Description

This PR reports stats for multistream downloads. With addition of copyTo() in PR proper stats are no longer reported. This PR fixes the issue and reports proper stats.

Related Issues

Resolves #10283

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
GitHub issue/PR created in OpenSearch documentation repo for the required public documentation changes (#[Issue/PR number])

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Rishikesh1159 <[email protected]>

Rishikesh1159 · 2023-10-05T14:05:47Z

This PR fixes the stats reporting for multistream downloads. I have tested the working of fix by following testing:

Testing:
-> Created a cluster with remote store enabled and repository registered.
-> Ingested some data and made call to _remotestore/stats api to get download stats.

-> Without this PR fix (download stats are empty for replicas):

{
 "_shards" : {
  "total" : 2,
  "successful" : 2,
  "failed" : 0
 },
 "indices" : {
  "nyc_taxis" : {
   "shards" : {
    "0" : [
     {
      "routing" : {
       "state" : "STARTED",
       "primary" : true,
       "node" : "eGJVITx2QAmhJ0Y6ywgKvw"
      },
      "segment" : {
       "download" : { },
       "upload" : {
        "local_refresh_timestamp_in_millis" : 1696445223665,
        "remote_refresh_timestamp_in_millis" : 1696445223665,
        "refresh_time_lag_in_millis" : 0,
        "refresh_lag" : 0,
        "bytes_lag" : 0,
        "backpressure_rejection_count" : 0,
        "consecutive_failure_count" : 0,
        "total_uploads" : {
         "started" : 129,
         "succeeded" : 129,
         "failed" : 0
        },
        "total_upload_size" : {
         "started_bytes" : 3756462697,
         "succeeded_bytes" : 3756462697,
         "failed_bytes" : 0
        },
        "remote_refresh_size_in_bytes" : {
         "last_successful" : 0,
         "moving_avg" : 7.698216395E7
        },
        "upload_speed_in_bytes_per_sec" : {
         "moving_avg" : 4.59313421E7
        },
        "remote_refresh_latency_in_millis" : {
         "moving_avg" : 829.4
        }
       }
      },
      "translog" : {
       "upload" : {
        "last_successful_upload_timestamp" : 1696445164677,
        "total_uploads" : {
         "started" : 554,
         "failed" : 0,
         "succeeded" : 554
        },
        "total_upload_size" : {
         "started_bytes" : 3308030813,
         "failed_bytes" : 0,
         "succeeded_bytes" : 3308030813
        },
        "total_upload_time_in_millis" : 209010433,
        "upload_size_in_bytes" : {
         "moving_avg" : 5061984.05
        },
        "upload_speed_in_bytes_per_sec" : {
         "moving_avg" : 7328.3
        },
        "upload_time_in_millis" : {
         "moving_avg" : 715100.5
        }
       },
       "download" : { }
      }
     },
     {
      "routing" : {
       "state" : "STARTED",
       "primary" : false,
       "node" : "W7neivyKQpiIxwffoZCXrw"
      },
      "segment" : {
       "download" : { },
       "upload" : { }
      },
      "translog" : {
       "upload" : { },
       "download" : { }
      }
     }
    ]
   }
  }
 }
}

-> With this PR fix (download stats for replica are reported):

curl localhost:9200/_remotestore/stats/nyc_taxis?pretty
{
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "indices" : {
    "nyc_taxis" : {
      "shards" : {
        "0" : [
          {
            "routing" : {
              "state" : "STARTED",
              "primary" : true,
              "node" : "2C-7q8IkR4aM4N1Uj1jsJA"
            },
            "segment" : {
              "download" : { },
              "upload" : {
                "local_refresh_timestamp_in_millis" : 1696446555174,
                "remote_refresh_timestamp_in_millis" : 1696446555174,
                "refresh_time_lag_in_millis" : 0,
                "refresh_lag" : 0,
                "bytes_lag" : 0,
                "backpressure_rejection_count" : 0,
                "consecutive_failure_count" : 0,
                "total_uploads" : {
                  "started" : 103,
                  "succeeded" : 103,
                  "failed" : 0
                },
                "total_upload_size" : {
                  "started_bytes" : 2589972453,
                  "succeeded_bytes" : 2589972453,
                  "failed_bytes" : 0
                },
                "remote_refresh_size_in_bytes" : {
                  "last_successful" : 0,
                  "moving_avg" : 5.309439115E7
                },
                "upload_speed_in_bytes_per_sec" : {
                  "moving_avg" : 4.399847085E7
                },
                "remote_refresh_latency_in_millis" : {
                  "moving_avg" : 822.1
                }
              }
            },
            "translog" : {
              "upload" : {
                "last_successful_upload_timestamp" : 1696446532704,
                "total_uploads" : {
                  "started" : 431,
                  "failed" : 0,
                  "succeeded" : 431
                },
                "total_upload_size" : {
                  "started_bytes" : 2596014317,
                  "failed_bytes" : 0,
                  "succeeded_bytes" : 2596014317
                },
                "total_upload_time_in_millis" : 127155781,
                "upload_size_in_bytes" : {
                  "moving_avg" : 5638599.25
                },
                "upload_speed_in_bytes_per_sec" : {
                  "moving_avg" : 10546.45
                },
                "upload_time_in_millis" : {
                  "moving_avg" : 536076.7
                }
              },
              "download" : { }
            }
          },
          {
            "routing" : {
              "state" : "STARTED",
              "primary" : false,
              "node" : "y6DkskhCRECaNG2zq2rF5A"
            },
            "segment" : {
              "download" : {
                "last_sync_timestamp" : 1696446709535,
                "total_download_size" : {
                  "started_bytes" : 2161626590,
                  "succeeded_bytes" : 2161626590,
                  "failed_bytes" : 0
                },
                "download_size_in_bytes" : {
                  "last_successful" : 337,
                  "moving_avg" : 1.18097258E7
                },
                "download_speed_in_bytes_per_sec" : {
                  "moving_avg" : 1.31509003E7
                }
              },
              "upload" : { }
            },
            "translog" : {
              "upload" : { },
              "download" : { }
            }
          }
        ]
      }
    }
  }
}

github-actions · 2023-10-05T14:19:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/27120/
CommitID: 3b6aba3
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-05T16:06:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/27127/
CommitID: 3b6aba3
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java

Signed-off-by: Rishikesh1159 <[email protected]>

github-actions · 2023-10-05T18:49:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/27162/
CommitID: 05f48e7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-05T19:40:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/27170/
CommitID: 05f48e7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-05T20:37:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/27178/
CommitID: 05f48e7

codecov · 2023-10-05T20:38:15Z

Codecov Report

Merging #10402 (05f48e7) into main (2965e69) will decrease coverage by 0.01%.
Report is 12 commits behind head on main.
The diff coverage is 79.45%.

@@             Coverage Diff              @@
##               main   #10402      +/-   ##
============================================
- Coverage     71.16%   71.15%   -0.01%     
- Complexity    58297    58375      +78     
============================================
  Files          4830     4843      +13     
  Lines        274903   275203     +300     
  Branches      40059    40068       +9     
============================================
+ Hits         195637   195830     +193     
- Misses        62882    62965      +83     
- Partials      16384    16408      +24

Files	Coverage Δ
...arch/telemetry/metrics/DefaultMetricsRegistry.java	`100.00% <100.00%> (ø)`
...ch/telemetry/metrics/noop/NoopMetricsRegistry.java	`100.00% <100.00%> (ø)`
...va/org/opensearch/telemetry/metrics/tags/Tags.java	`100.00% <100.00%> (ø)`
...org/opensearch/repositories/url/URLRepository.java	`72.00% <100.00%> (ø)`
...g/opensearch/transport/netty4/Netty4Transport.java	`73.65% <100.00%> (ø)`
...opensearch/repositories/azure/AzureRepository.java	`68.88% <100.00%> (-1.33%)`	⬇️
...g/opensearch/repositories/hdfs/HdfsRepository.java	`61.90% <100.00%> (ø)`
...java/org/opensearch/repositories/s3/S3Service.java	`75.52% <100.00%> (+0.52%)`	⬆️
.../org/opensearch/telemetry/OTelTelemetryPlugin.java	`100.00% <100.00%> (ø)`
...rg/opensearch/telemetry/OTelTelemetrySettings.java	`100.00% <100.00%> (ø)`
... and 41 more

... and 492 files with indirect coverage changes

* Fix stats reporting for multistream downloads. Signed-off-by: Rishikesh1159 <[email protected]> * rename tracker to fileTransferTracker. Signed-off-by: Rishikesh1159 <[email protected]> --------- Signed-off-by: Rishikesh1159 <[email protected]> (cherry picked from commit a3f8432) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

#10417) * Fix stats reporting for multistream downloads. * rename tracker to fileTransferTracker. --------- (cherry picked from commit a3f8432) Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

#10424) * Fix stats reporting for multistream downloads. * rename tracker to fileTransferTracker. --------- (cherry picked from commit a3f8432) Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…arch-project#10402) * Fix stats reporting for multistream downloads. Signed-off-by: Rishikesh1159 <[email protected]> * rename tracker to fileTransferTracker. Signed-off-by: Rishikesh1159 <[email protected]> --------- Signed-off-by: Rishikesh1159 <[email protected]>

…arch-project#10402) * Fix stats reporting for multistream downloads. Signed-off-by: Rishikesh1159 <[email protected]> * rename tracker to fileTransferTracker. Signed-off-by: Rishikesh1159 <[email protected]> --------- Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

Fix stats reporting for multistream downloads.

3b6aba3

Signed-off-by: Rishikesh1159 <[email protected]>

github-actions bot added distributed framework enhancement Enhancement or improvement to existing feature or request Indexing:Replication Issues and PRs related to core replication framework eg segrep labels Oct 5, 2023

Rishikesh1159 mentioned this pull request Oct 5, 2023

[Remote Store] Fix Report Stats for MultiStream Downloads #10357

Closed

7 tasks

kotwanikunal added skip-changelog backport 2.x Backport to 2.x branch v2.11.0 Issues and PRs related to version 2.11.0 labels Oct 5, 2023

andrross reviewed Oct 5, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java Show resolved Hide resolved

rename tracker to fileTransferTracker.

05f48e7

Signed-off-by: Rishikesh1159 <[email protected]>

peterzhuamazon mentioned this pull request Oct 5, 2023

[RELEASE] Release version 2.11.0 opensearch-project/opensearch-build#3998

Closed

71 tasks

kotwanikunal approved these changes Oct 5, 2023

View reviewed changes

Rishikesh1159 merged commit a3f8432 into opensearch-project:main Oct 5, 2023
12 of 13 checks passed

opensearch-trigger-bot bot mentioned this pull request Oct 5, 2023

[Backport 2.x] [Remote Store] Fix stats reporting for multistream downloads. #10417

Merged

kotwanikunal added the backport 2.11 label Oct 5, 2023

opensearch-trigger-bot bot mentioned this pull request Oct 5, 2023

[Backport 2.11] [Remote Store] Fix stats reporting for multistream downloads. #10424

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Fix stats reporting for multistream downloads. #10402

[Remote Store] Fix stats reporting for multistream downloads. #10402

Rishikesh1159 commented Oct 5, 2023

Rishikesh1159 commented Oct 5, 2023 •

edited

Loading

github-actions bot commented Oct 5, 2023

github-actions bot commented Oct 5, 2023

github-actions bot commented Oct 5, 2023

github-actions bot commented Oct 5, 2023

github-actions bot commented Oct 5, 2023

codecov bot commented Oct 5, 2023

[Remote Store] Fix stats reporting for multistream downloads. #10402

[Remote Store] Fix stats reporting for multistream downloads. #10402

Conversation

Rishikesh1159 commented Oct 5, 2023

Description

Related Issues

Check List

Rishikesh1159 commented Oct 5, 2023 • edited Loading

github-actions bot commented Oct 5, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 5, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 5, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 5, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 5, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Oct 5, 2023

Codecov Report

Rishikesh1159 commented Oct 5, 2023 •

edited

Loading